A Multimodal Dialogue Contribution Coding Scheme

In this paper a coding scheme for annotating speech acts in the context of multimodal design tasks is presented. The scheme is an extension of the dialogue act markup in several layers scheme DAMSL (Allen and Core, 1997). This scheme has been used to annotate human-human task oriented conversations; however, information other than spoken language, like deictic gestures or information conveyed through external representations, such as paper or graphical screen, cannot be taken into account with DAMSL and related coding schemes. The extension proposed provides a methodology to capture deictic and graphical information common in design task oriented dialogues. The scheme is been developed in the context of the Diálogo Inteligente Multimodal en Español program (DIME) to support Spanish spoken human-computer conversations in which the system has the role of a design assistant in the kitchen design domain.